Майнор "Интеллектуальный анализ данных"

Курс "Введение в анализ данных"

Код

Текст

Лабораторная работа №3. Supervised Learning

Данные

Код

Текст

В рамках данной лабораторной работы вам предлагается проанализировать набор данных о студентах двух школ в Португалии.
В файле students_data.csv представлена информация о студентах, посещающих два курса - математику (Math) и поргутальский язык (Por). Некоторые студенты представлены в обоих курсах, некоторые - только в одном. Для каждого студента известны три оценки по курсу: оценка за первое полугодие (G1), оценка за второе полугодие (G2) и итоговая оценка за год (G3).

Код

Текст

Requirement already satisfied: graphviz in /usr/local/lib/python3.7/dist-packages (0.10.1)
Requirement already satisfied: wand in /usr/local/lib/python3.7/dist-packages (0.6.6)
Reading package lists... Done
Building dependency tree       
Reading state information... Done
libmagickwand-dev is already the newest version (8:6.9.7.4+dfsg-16ubuntu6.9).
0 upgraded, 0 newly installed, 0 to remove and 39 not upgraded.
Requirement already satisfied: xgboost in /usr/local/lib/python3.7/dist-packages (0.90)
Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from xgboost) (1.19.5)
Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from xgboost) (1.4.1)

Код

Текст

Enter to Rename, ⇧Enter to Preview

Код

Текст

Enter to Rename, ⇧Enter to Preview

(1044, 36)

Enter to Rename, ⇧Enter to Preview

Признаки

Данные представлены признаками различных типов: числовыми, категориальными, упорядоченными категориальными.

Описание признаков:

Enter to Rename, ⇧Enter to Preview

Часть 1. Предобработка данных

Разделите данные на две части - данные для моделирования (80%) и отложенную выборку (20%). Убедитесь, что распределение целевой переменной (G3) одинаково в обоих частях.
NB: Отложенную выборку нужно использовать только для финальной оценки качества модели. Обучение и кросс-валидацию следует проводить на данных для моделирования.
Выполните необходимые преобразования данных: исправление ошибок, удаление выбросов и пропусков, приведение признаков к числовому виду.
Оцените значимость признаков для определения итоговой оценки за курс. Исключите из выборки незначимые на ваш взгляд признаки, обоснуйте свое решение.
(Опционально) Feature engineering: создайте новые признаки (значимые) на основе уже имеющихся.

Tip: Используйте свои наработки из Лабораторной работы №1.

Преобразование данных

Проверить на ошибочные значения

# Check mistakes in categorical values
def count_categories(data_frame=data, target_type = 'object', drop_id = True):
    categories = {label : set(data_frame[label].values) for label in data_frame.columns 
                  if data_frame[label].dtypes == target_type and not (drop_id and label == "ID")}
    for key in categories:
        categories[key] = (categories[key], len(categories[key]))
    # Let's watch what categorical values we have in each column
    return categories
# return categories
print_dict(count_categories())

Enter to Rename, ⇧Enter to Preview

Key         Item
----------  ---------------------------------------------------------------------
Subject     ({'Math', 'Por'}, 2)
school      ({'MS', 'GP'}, 2)
sex         ({'m', 'F', 'M'}, 3)
address     ({'U', 'R'}, 2)
famsize     ({'GT3', 'LE3'}, 2)
Pstatus     ({'t', 'A', 'T'}, 3)
Medu        ({'1', '3', '2', 'o', '4', '0'}, 6)
Fedu        ({'1', '3', '2', 'o', '4', '0'}, 6)
Mjob        ({'services', 'teacher', 'at-home', 'other', 'health', 'at_home'}, 6)
Fjob        ({'services', 'teacher', 'at-home', 'other', 'health', 'at_home'}, 6)
reason      ({'reputation', 'other', 'home', 'course'}, 4)
guardian    ({'futher', 'other', 'mother', 'father'}, 4)
schoolsup   ({'yes', 'no'}, 2)
famsup      ({'yes', 'no'}, 2)
paid        ({'yes', 'no'}, 2)
activities  ({'yes', 'no'}, 2)
nursery     ({'yes', 'no'}, 2)
higher      ({'yes', 'no'}, 2)
internet    ({'yes', 'no'}, 2)
romantic    ({nan, 'yes', 'no'}, 3)
cheating    ({nan, 'yes', 'no'}, 3)

# Columns with errors:
# sex ('m'), Pstatus ('t'), Medu and Fedu ('0' and strings instead int64)),
# Mjob and Fjob ('at-home' = 'at_home'), guardian ('futher'), 
# romantic and cheating have nan - I'll fix it later
#
data['sex'].replace('m', 'M', inplace=True)
data['Pstatus'].replace('t', 'T', inplace=True)
data['Medu'].replace('o', '0', inplace=True)
data['Fedu'].replace('o', '0', inplace=True)
data['Mjob'].replace('at-home', 'at_home', inplace=True)
data['Fjob'].replace('at-home', 'at_home', inplace=True)
data['guardian'].replace('futher', 'father', inplace=True)
print_dict(count_categories())

Enter to Rename, ⇧Enter to Preview

Key         Item
----------  ----------------------------------------------------------
Subject     ({'Math', 'Por'}, 2)
school      ({'MS', 'GP'}, 2)
sex         ({'F', 'M'}, 2)
address     ({'U', 'R'}, 2)
famsize     ({'GT3', 'LE3'}, 2)
Pstatus     ({'A', 'T'}, 2)
Medu        ({'1', '3', '2', '4', '0'}, 5)
Fedu        ({'1', '3', '2', '4', '0'}, 5)
Mjob        ({'services', 'teacher', 'other', 'health', 'at_home'}, 5)
Fjob        ({'services', 'teacher', 'other', 'health', 'at_home'}, 5)
reason      ({'reputation', 'other', 'home', 'course'}, 4)
guardian    ({'other', 'mother', 'father'}, 3)
schoolsup   ({'yes', 'no'}, 2)
famsup      ({'yes', 'no'}, 2)
paid        ({'yes', 'no'}, 2)
activities  ({'yes', 'no'}, 2)
nursery     ({'yes', 'no'}, 2)
higher      ({'yes', 'no'}, 2)
internet    ({'yes', 'no'}, 2)
romantic    ({nan, 'yes', 'no'}, 3)
cheating    ({nan, 'yes', 'no'}, 3)

Enter to Rename, ⇧Enter to Preview

Medu: int64 , Fedu: int64

Проверить на пропущенные значения

Enter to Rename, ⇧Enter to Preview

romantic     10
famrel       10
Dalc         10
Walc         10
cheating    700
dtype: int64

Код

Текст

Enter to Rename, ⇧Enter to Preview

no     664
yes    370
NaN     10
Name: romantic, dtype: int64
4.0    508
5.0    283
3.0    167
2.0     46
1.0     30
NaN     10
Name: famrel, dtype: int64
1.0    719
2.0    194
3.0     69
5.0     26
4.0     26
NaN     10
Name: Dalc, dtype: int64
1.0    393
2.0    231
3.0    199
4.0    138
5.0     73
NaN     10
Name: Walc, dtype: int64
NaN    700
yes    175
no     169
Name: cheating, dtype: int64

Enter to Rename, ⇧Enter to Preview

Код

Текст

Enter to Rename, ⇧Enter to Preview

famrel:  3.9361702127659575
Dalc:  1.4970986460348163
Walc:  2.2911025145067696

Enter to Rename, ⇧Enter to Preview

Код

Текст

Основываясь на предыдующей работе, даатсет не содержит серьезных выбросов, способных крайне негативно повлиять на построение моделей

Категориальные признаки нужно привести к One-Hot-Encoded форме

Код

Текст

# Label encoding
object_labels = [label for label in data.columns if data[label].dtypes == 'object']
data_labels_enc = data[object_labels].copy()
 
cats_info = count_categories(data_labels_enc)
for label in data_labels_enc.columns:
    cats_info[label] = (tuple(cats_info[label][0]), cats_info[label][1])
    for i in range(cats_info[label][1]):
        data_labels_enc[label].replace(cats_info[label][0][i], i, inplace=True)
 
data_labels_enc.head(5)

Enter to Rename, ⇧Enter to Preview

# One hot encoding for several catigorical columns
enc = OneHotEncoder(handle_unknown='ignore')
 
ohenc_labels = ['Mjob', 'Fjob', 'guardian', 'reason']
ohenc_data = pd.DataFrame(enc.fit_transform(data_labels_enc[ohenc_labels]).toarray())
ohenc_data.columns = ['is_' + label + '_'+ str(cats_info[label][0][i]) 
                     for label in ohenc_labels
                     for i in range(cats_info[label][1])]
for label in ohenc_data.columns:
    ohenc_data[label] = ohenc_data[label].apply(int, convert_dtype=True)
ohenc_data.head(5)

Enter to Rename, ⇧Enter to Preview

Learning data

Ground Truth значения

Enter to Rename, ⇧Enter to Preview

Данные для обучения

Enter to Rename, ⇧Enter to Preview

Без G1 признака

Enter to Rename, ⇧Enter to Preview

Learning data features (no G1):  44

Лишние признаки & Feature eng.

Посмотрим какие признаки сильно коррелируют между собой, ведь это может ухудшить результаты обучения

Выведем признаки с высокой корреляцией

Код

Текст

def find_high_cov(data, cov_level=0.4) -> dict:
    high_cov = dict()
    for c in range(len(data.columns)):
        for i in range(c + 1, len(data.columns)):
            label_c = data.columns[c]
            label_i = data.columns[i]
            value = data.loc[label_c, label_i]
            if abs(value) >= cov_level:
                high_cov[(label_c, label_i)] = value
    return high_cov
print_dict(find_high_cov(learning_data.corr()), 'Pair', 'Cov')

Enter to Rename, ⇧Enter to Preview

Pair                                                Cov
--------------------------------------------  ---------
('Medu', 'Fedu')                               0.642063
('Medu', 'is_Mjob_teacher')                    0.452952
('Dalc', 'Walc')                               0.626551
('is_Subject_Por', 'is_paid_no')               0.473453
('is_Mjob_services', 'is_Mjob_other')         -0.428556
('is_Fjob_services', 'is_Fjob_other')         -0.702118
('is_guardian_other', 'is_guardian_mother')   -0.416173
('is_guardian_mother', 'is_guardian_father')  -0.836005
('is_reason_reputation', 'is_reason_course')  -0.46711
('is_reason_home', 'is_reason_course')        -0.479456

Многие из коррелирующих признаков - взаимоисключающие one-hot encoded признаки и их изменять не стоит

Однако я обрачу внимание на взаимосвязь признаков Medu и Fedu, Dalc и Walc
В данном случае эти 4 признака можно свести к 2м путем объединения:

Medu + Fedu = Pedu (можно считать образование обоих родителей)
Walc + Dalc = Alc (учитываем общее пристрастие к алкоголю)

Enter to Rename, ⇧Enter to Preview

Split data

Enter to Rename, ⇧Enter to Preview

Код

Текст

установлен random_state чтобы контролировать разбиение

Код

Текст

Enter to Rename, ⇧Enter to Preview

Код

Текст

Убедимся в правильности разбиения (в частности соотвествии y)

Код

Текст

Enter to Rename, ⇧Enter to Preview

860    11
269     9
581     0
775    11
669     9
Name: G3, dtype: int64

Код

Текст

Рассмотрим распределение G3 в выборках y_train и y_test

Код

Текст

Код

Текст

Мы можем видеть, что распределение G3 в y_train и y_test выборках одинаково

Код

Текст

Scale data

Код

Текст

Enter to Rename, ⇧Enter to Preview

Код

Текст

Enter to Rename, ⇧Enter to Preview

Код

Текст

Enter to Rename, ⇧Enter to Preview

Код

Текст

Код

Текст

Enter to Rename, ⇧Enter to Preview

Train shape:  (835, 43)
Example:

array([ 1.77837403, -0.72568138,  0.0362296 , -0.40926236,  0.05351395,
       -0.18409633, -1.8956647 , -1.77756101,  1.19804442, -0.03615913,
       -1.25268941,  0.61472578,  1.13199857, -0.61657545,  1.59787092,
       -2.72659924,  0.36042375, -0.80837203,  0.53534546,  1.00359929,
       -0.49437974, -0.3095166 , -0.50747426,  0.75468462,  1.90085761,
       -0.37303373, -0.79226278, -0.30718781, -0.48309685, -0.62583278,
       -0.25505102, -1.1430791 , -0.19962605,  4.23996141, -0.27573706,
        0.65558716, -0.54644333,  1.78773038, -0.34322668, -0.5925645 ,
       -0.81039869, -0.88058892,  1.00376842])

Код

Текст

Enter to Rename, ⇧Enter to Preview

Test shape:  (209, 43)
Example:

array([ 0.20116499, -0.72568138,  0.0362296 , -0.40926236,  1.14337127,
        0.77068897, -1.01432838,  1.03887816, -0.70439696,  1.62279005,
       -1.25268941,  0.61472578, -0.88339334, -0.61657545, -0.62583278,
        0.36675724,  0.36042375, -0.80837203,  0.53534546, -0.99641362,
       -0.49437974, -0.3095166 ,  1.97054331,  0.75468462,  1.90085761,
       -0.37303373, -0.79226278, -0.30718781, -0.48309685,  1.59787092,
       -0.25505102, -1.1430791 , -0.19962605, -0.2358512 , -0.27573706,
       -1.5253502 ,  1.83001594,  1.78773038, -0.34322668, -0.5925645 ,
       -0.81039869,  0.63756335,  0.51016027])

Код

Текст

Часть 2. Регрессия

Код

Текст

Решите задачу регрессии: постройте модель, предсказывающую итоговую оценку, которую получит студент по предмету (G3). При решении задачи нельзя использовать признак G2.
Для решения задачи примените следующие методы:
- Линейная регрессия + регуляризации
- Полиномиальная регрессия
- KNN
- Деревья решений, Random Forest
Для каждого метода выполните настройку гиперпараметров на кросс-валидации.
Оцените качество каждой модели на отложенной выборке, используйте различные метрики. Сравните модели и сделайте вывод о качестве решения задачи.
Задачу необходимо решить в двух вариантах: с использованием признака G1 и без него. Сравните качество решений в двух случаях.
В регрессионных моделях попробуйте дать интерпретацию весам признаков.

Enter to Rename, ⇧Enter to Preview

Код

Текст

def reg_scores(predicted, labels, printable=False):
  scores = dict()
  scores['mse'] = metrics.mean_squared_error(labels, predicted)
  scores['rmse'] = np.sqrt(scores['mse'])
  scores['mae'] = metrics.mean_absolute_error(labels, predicted)
  scores['medae'] = metrics.median_absolute_error(labels, predicted)
  scores['R2'] = metrics.r2_score(labels, predicted)
  # print
  if printable:
    scores_table = [(key, value) for key, value in scores.items()]
    print('SCORES:')
    print(tb(scores_table, headers=('metric', 'score'), tablefmt="fancy_grid"))
  return scores

Код

Текст

def reg_scores_compare(predicted: dict, labels: dict, printable=True):
  scores_train = reg_scores(predicted['train'], labels['train'])
  scores_test = reg_scores(predicted['test'], labels['test'])
  # print
  if printable:
    keys = scores_train.keys()
    train_values = scores_train.values()
    test_values = scores_test.values()
    scores_table = [(key, train, test) for key, train, test in 
                    zip(keys, train_values, test_values)]
    print('SCORES:')
    print(tb(scores_table, 
             headers=('metric', 'train scores', 'test scores'), 
             tablefmt="fancy_grid"))            
  return scores_train, scores_test

def create_table_from_series(series, table_width=4):
  row_counter = 0
  row = []
  table = []
  for i in range(series.size):
    row.append(i)
    row.append(series.index[i])
    row.append("{:.3f}".format(series[i]))
 
    row_counter += 1
    if (row_counter + 1) % (table_width + 1) == 0:
      table.append(row)
      row = []
      row_counter = 0
  return table

def create_columns_from_series(series, table_width=4):
  row_counter = 0
  row = []
  table = []
  for i in range(series.size):
    row.append(i)
    row.append(series.index[i])
    row.append("{:.3f}".format(series[i]))
 
    row_counter += 1
    if (row_counter + 1) % (table_width + 1) == 0:
      table.append(row)
      row = []
      row_counter = 0
  if len(row):
    for i in range((table_width - row_counter) * 2):
      row.append(' ')
    table.append(row)
  return table

Enter to Rename, ⇧Enter to Preview

Linear regression

Код

Текст

Enter to Rename, ⇧Enter to Preview

Losso CV

Enter to Rename, ⇧Enter to Preview

Alpha:  0.1
CPU times: user 248 ms, sys: 212 ms, total: 460 ms
Wall time: 242 ms

Test

Код

Текст

y_predicted_train = losso_model.predict(x_train)
y_predicted_test  = losso_model.predict(x_test)
 
_, reg_test_scores['losso'] = reg_scores_compare({'train': y_predicted_train, 
                                                  'test': y_predicted_test},
                                                 {'train': y_train, 
                                                  'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │      13.6048   │    11.5461    │
├──────────┼────────────────┼───────────────┤
│ rmse     │       3.68847  │     3.39796   │
├──────────┼────────────────┼───────────────┤
│ mae      │       2.66373  │     2.41408   │
├──────────┼────────────────┼───────────────┤
│ medae    │       1.89809  │     1.70175   │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.116701 │     0.0880885 │
╘══════════╧════════════════╧═══════════════╛

Enter to Rename, ⇧Enter to Preview

WEIGHTS
G1                      0.295871
is_Subject_Math        -0.000000
is_paid_yes            -0.000000
is_famsup_yes           0.000000
is_schoolsup_yes       -0.000000
is_Pstatus_T           -0.000000
is_famsize_GT3         -0.000000
is_address_U            0.000000
is_sex_M               -0.000000
is_school_GP            0.000000
Pedu                    0.000000
is_nursery_yes          0.000000
absences               -0.000000
health                 -0.000000
goout                  -0.000000
freetime               -0.000000
famrel                  0.000000
failures               -0.000000
studytime               0.000000
traveltime             -0.000000
is_activities_yes       0.000000
is_higher_yes           0.000000
Alc                    -0.000000
is_Fjob_services       -0.000000
is_reason_home          0.000000
is_reason_other        -0.000000
is_reason_reputation    0.000000
is_reason_course       -0.000000
is_guardian_father      0.000000
is_guardian_other      -0.000000
is_guardian_mother     -0.000000
is_Fjob_other          -0.000000
is_Fjob_at_home        -0.000000
is_internet_yes         0.000000
is_Fjob_teacher         0.000000
is_Fjob_health          0.000000
is_Mjob_other          -0.000000
is_Mjob_services        0.000000
is_Mjob_at_home        -0.000000
is_Mjob_teacher         0.000000
is_Mjob_health          0.000000
is_romantic_yes        -0.000000
age                    -0.000000
dtype: float64

Метрики у модели очень слабые, а по весам можно видеть, что предсказание делается, основываясь только на G1 признаке - предыдущей оценке

No G1

Enter to Rename, ⇧Enter to Preview

Alpha:  29.900000000000002
CPU times: user 258 ms, sys: 223 ms, total: 481 ms
Wall time: 259 ms

Test

y_predicted_train = losso_model_nog1.predict(x_no_g1_train)
y_predicted_test  = losso_model_nog1.predict(x_no_g1_test)
 
_, reg_test_scores_nog1['losso'] = reg_scores_compare({'train': y_predicted_train, 
                                                       'test': y_predicted_test},
                                                      {'train': y_train, 
                                                       'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │       15.4023  │    13.0908    │
├──────────┼────────────────┼───────────────┤
│ rmse     │        3.92457 │     3.61813   │
├──────────┼────────────────┼───────────────┤
│ mae      │        2.8788  │     2.61368   │
├──────────┼────────────────┼───────────────┤
│ medae    │        2.21078 │     1.78922   │
├──────────┼────────────────┼───────────────┤
│ R2       │        0       │    -0.0339102 │
╘══════════╧════════════════╧═══════════════╛

Enter to Rename, ⇧Enter to Preview

WEIGHTS
Pedu                    0.0
is_school_GP            0.0
is_activities_yes       0.0
is_paid_yes            -0.0
is_famsup_yes          -0.0
is_schoolsup_yes       -0.0
is_Pstatus_T           -0.0
is_famsize_GT3         -0.0
is_address_U            0.0
is_sex_M               -0.0
is_Subject_Math        -0.0
Alc                    -0.0
absences               -0.0
health                 -0.0
goout                  -0.0
freetime               -0.0
famrel                  0.0
failures               -0.0
studytime               0.0
traveltime             -0.0
is_nursery_yes          0.0
is_higher_yes           0.0
is_internet_yes         0.0
is_romantic_yes        -0.0
is_reason_home          0.0
is_reason_other        -0.0
is_reason_reputation    0.0
is_reason_course       -0.0
is_guardian_father      0.0
is_guardian_other      -0.0
is_guardian_mother     -0.0
is_Fjob_other          -0.0
is_Fjob_services       -0.0
is_Fjob_at_home        -0.0
is_Fjob_teacher         0.0
is_Fjob_health          0.0
is_Mjob_other          -0.0
is_Mjob_services        0.0
is_Mjob_at_home        -0.0
is_Mjob_teacher         0.0
is_Mjob_health          0.0
age                    -0.0
dtype: float64

Убрав у модели G1 признак мы вовсе лешили ее возможности предсказывать - все веса равны нулю

Ridge CV

Код

Текст

Enter to Rename, ⇧Enter to Preview

Alpha:  28.6
CPU times: user 5.68 s, sys: 7.22 s, total: 12.9 s
Wall time: 6.69 s

Код

Текст

Test

Код

Текст

y_predicted_train = ridge_model.predict(x_train)
y_predicted_test  = ridge_model.predict(x_test)
 
_, reg_test_scores['ridge'] = reg_scores_compare({'train': y_predicted_train, 
                                                  'test': y_predicted_test},
                                                 {'train': y_train, 
                                                  'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │       4.60432  │       4.91063 │
├──────────┼────────────────┼───────────────┤
│ rmse     │       2.14577  │       2.21599 │
├──────────┼────────────────┼───────────────┤
│ mae      │       1.42811  │       1.59266 │
├──────────┼────────────────┼───────────────┤
│ medae    │       0.947277 │       1.27257 │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.701062 │       0.61216 │
╘══════════╧════════════════╧═══════════════╛

Enter to Rename, ⇧Enter to Preview

WEIGHTS
G1                      2.835659
is_Subject_Math        -0.618114
failures               -0.368477
health                 -0.228059
absences                0.218612
is_paid_yes             0.197687
is_romantic_yes        -0.196752
goout                  -0.176093
famrel                  0.158452
is_Pstatus_T           -0.135200
is_address_U            0.126000
is_higher_yes           0.116390
is_Mjob_health          0.109009
traveltime              0.093330
Alc                     0.088374
is_Mjob_services        0.074440
is_famsize_GT3         -0.072063
is_Fjob_services       -0.066631
is_Mjob_at_home        -0.060031
is_Mjob_other          -0.055127
is_reason_home         -0.054204
is_school_GP            0.053477
is_nursery_yes         -0.052833
is_Fjob_other           0.050603
is_reason_course        0.048157
is_schoolsup_yes        0.042978
freetime               -0.035239
is_internet_yes         0.034970
is_Mjob_teacher        -0.033365
is_guardian_father      0.032680
is_activities_yes      -0.028371
is_guardian_mother     -0.027458
is_sex_M                0.020741
is_Fjob_health          0.019942
is_Fjob_at_home         0.017033
Pedu                   -0.013755
age                    -0.012560
is_Fjob_teacher        -0.011470
studytime               0.010736
is_famsup_yes           0.006827
is_reason_other        -0.004952
is_guardian_other      -0.004532
is_reason_reputation    0.004077
dtype: float64

Данная линейная модель с Ridge регуляризацией показала хорошие результаты, судя по метрикам.

Также по весам можно понять, что модель использует все параметры для предсказания в разной мере.
Наиболее влияетельными оказались:

предыдущая оценка G1 (не удивительно)
предмет is_subject_Math (отрицательный вес показывает, что по математике оценка должны быть ниже)
прогулы failures (больше пропускаешь - оценка ниже)

No G1

Enter to Rename, ⇧Enter to Preview

Alpha:  29.900000000000002
CPU times: user 5.74 s, sys: 7.03 s, total: 12.8 s
Wall time: 6.62 s

Test

y_predicted_train = ridge_model_nog1.predict(x_no_g1_train)
y_predicted_test  = ridge_model_nog1.predict(x_no_g1_test)
 
_, reg_test_scores_nog1['ridge'] = reg_scores_compare({'train': y_predicted_train, 
                                                       'test': y_predicted_test},
                                                      {'train': y_train, 
                                                       'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │      10.704    │     11.0084   │
├──────────┼────────────────┼───────────────┤
│ rmse     │       3.2717   │      3.3179   │
├──────────┼────────────────┼───────────────┤
│ mae      │       2.38736  │      2.41881  │
├──────────┼────────────────┼───────────────┤
│ medae    │       1.72638  │      1.87332  │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.305035 │      0.130556 │
╘══════════╧════════════════╧═══════════════╛

Enter to Rename, ⇧Enter to Preview

WEIGHTS
failures               -1.065998
is_Subject_Math        -0.947771
is_higher_yes           0.482705
studytime               0.427825
is_schoolsup_yes       -0.393510
health                 -0.388848
is_school_GP            0.329083
goout                  -0.306595
is_Mjob_health          0.286244
is_romantic_yes        -0.273986
is_famsize_GT3         -0.247504
famrel                  0.239709
is_address_U            0.237777
is_famsup_yes          -0.231084
is_Fjob_teacher         0.221638
is_Mjob_other          -0.192995
is_Mjob_services        0.190275
is_Mjob_at_home        -0.157597
Pedu                    0.146618
Alc                    -0.137913
is_Fjob_services       -0.123705
is_paid_yes             0.116387
absences                0.114731
is_internet_yes         0.108024
is_nursery_yes         -0.100919
is_guardian_mother     -0.088158
is_guardian_other       0.077666
is_Pstatus_T           -0.073282
traveltime              0.055811
is_activities_yes       0.052456
freetime                0.051664
is_reason_reputation    0.051609
is_guardian_father      0.048765
age                    -0.042837
is_Fjob_at_home         0.042574
is_sex_M                0.035327
is_reason_other        -0.031182
is_reason_home         -0.021044
is_Fjob_other          -0.010299
is_Fjob_health         -0.009660
is_Mjob_teacher        -0.009414
is_reason_course       -0.006510
dtype: float64

Без признака G1 метрики стали сильно хуже, а большее влияение получили:

failures
is_higher_yes
studytime

Код

Текст

Polynomial

Enter to Rename, ⇧Enter to Preview

CPU times: user 190 ms, sys: 50.9 ms, total: 241 ms
Wall time: 13.9 s

GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),
             error_score=nan,
             estimator=Pipeline(memory=None,
                                steps=[('polynomialfeatures',
                                        PolynomialFeatures(degree=2,
                                                           include_bias=False,
                                                           interaction_only=False,
                                                           order='C')),
                                       ('linearregression',
                                        LinearRegression(copy_X=True,
                                                         fit_intercept=True,
                                                         n_jobs=None,
                                                         normalize=True))],
                                verbose=False),
             iid='deprecated', n_jobs=-1,
             param_grid={'polynomialfeatures__degree': [1, 2, 3]},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring='neg_mean_squared_error', verbose=0)

Best hyperparams

Enter to Rename, ⇧Enter to Preview

{'polynomialfeatures__degree': 1}

Enter to Rename, ⇧Enter to Preview

Test

y_predicted_train = polynomial_model.predict(x_train)
y_predicted_test  = polynomial_model.predict(x_test)
 
_, reg_test_scores['poly'] = reg_scores_compare({'train': y_predicted_train, 
                                                  'test': y_predicted_test},
                                                 {'train': y_train, 
                                                  'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │       4.63996  │      5.02588  │
├──────────┼────────────────┼───────────────┤
│ rmse     │       2.15406  │      2.24185  │
├──────────┼────────────────┼───────────────┤
│ mae      │       1.45921  │      1.62587  │
├──────────┼────────────────┼───────────────┤
│ medae    │       0.973973 │      1.2604   │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.698748 │      0.603057 │
╘══════════╧════════════════╧═══════════════╛

Код

Текст

Давайте убедимся в том, что выбранная степень полинома оптимальна.
Поставим degree=3 принудительно и сравним метрики

poly_scaler.degree = 3
polynomial_model = make_pipeline(poly_scaler, reg_model)
polynomial_model.fit(x_train, y_train)
 
y_predicted_train = polynomial_model.predict(x_train)
y_predicted_test  = polynomial_model.predict(x_test)
 
_ = reg_scores_compare({'train': y_predicted_train, 
                        'test': y_predicted_test},
                       {'train': y_train, 
                        'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │    1.79099e-28 │       7.14285 │
├──────────┼────────────────┼───────────────┤
│ rmse     │    1.33828e-14 │       2.67261 │
├──────────┼────────────────┼───────────────┤
│ mae      │    1.03539e-14 │       1.97092 │
├──────────┼────────────────┼───────────────┤
│ medae    │    8.88178e-15 │       1.54451 │
├──────────┼────────────────┼───────────────┤
│ R2       │    1           │       0.43586 │
╘══════════╧════════════════╧═══════════════╛

На метриках мы видим переобучение модели:

на x_train выборке все метрики показывает минимально возможную ошибку, значит наша модель смогла построить функцию, чтобы "попасть" во все точки датасета
на x_test выборке модель показывает плохую обобщающую способность, по крайней мере хуже обычной линейной модели

No G1

Enter to Rename, ⇧Enter to Preview

CPU times: user 130 ms, sys: 17.4 ms, total: 147 ms
Wall time: 10.6 s

GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),
             error_score=nan,
             estimator=Pipeline(memory=None,
                                steps=[('polynomialfeatures',
                                        PolynomialFeatures(degree=2,
                                                           include_bias=False,
                                                           interaction_only=False,
                                                           order='C')),
                                       ('linearregression',
                                        LinearRegression(copy_X=True,
                                                         fit_intercept=True,
                                                         n_jobs=None,
                                                         normalize=True))],
                                verbose=False),
             iid='deprecated', n_jobs=-1,
             param_grid={'polynomialfeatures__degree': [1, 2, 3]},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring='neg_mean_squared_error', verbose=0)

Enter to Rename, ⇧Enter to Preview

{'polynomialfeatures__degree': 1}

Enter to Rename, ⇧Enter to Preview

Test

y_predicted_train = polynomial_model_nog1.predict(x_no_g1_train)
y_predicted_test  = polynomial_model_nog1.predict(x_no_g1_test)
 
_, reg_test_scores_nog1['poly'] = reg_scores_compare({'train': y_predicted_train, 
                                                      'test': y_predicted_test},
                                                     {'train': y_train, 
                                                      'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │      10.698    │      11.1208  │
├──────────┼────────────────┼───────────────┤
│ rmse     │       3.27078  │       3.33479 │
├──────────┼────────────────┼───────────────┤
│ mae      │       2.39404  │       2.43725 │
├──────────┼────────────────┼───────────────┤
│ medae    │       1.73181  │       1.89116 │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.305429 │       0.12168 │
╘══════════╧════════════════╧═══════════════╛

Метрики сильно хуже моделей с G1 признаком

Давайте убедимся в том, что выбранная степень полинома оптимальна.
Поставим degree=3 принудительно и сравним метрики

Код

Текст

poly_scaler.degree = 3
polynomial_model_nog1 = make_pipeline(poly_scaler, reg_model)
polynomial_model_nog1.fit(x_no_g1_train, y_train)
 
y_predicted_train = polynomial_model_nog1.predict(x_no_g1_train)
y_predicted_test  = polynomial_model_nog1.predict(x_no_g1_test)
 
_, reg_test_scores_nog1['poly'] = reg_scores_compare({'train': y_predicted_train, 
                                                      'test': y_predicted_test},
                                                     {'train': y_train, 
                                                      'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │    2.20209e-28 │    13.0745    │
├──────────┼────────────────┼───────────────┤
│ rmse     │    1.48395e-14 │     3.61587   │
├──────────┼────────────────┼───────────────┤
│ mae      │    1.16314e-14 │     2.66703   │
├──────────┼────────────────┼───────────────┤
│ medae    │    8.88178e-15 │     2.05421   │
├──────────┼────────────────┼───────────────┤
│ R2       │    1           │    -0.0326215 │
╘══════════╧════════════════╧═══════════════╛

KNN

Код

Текст

Код

Текст

Код

Текст

Код

Текст

CPU times: user 1.7 s, sys: 41.2 ms, total: 1.74 s
Wall time: 12.2 s

GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),
             error_score=nan,
             estimator=Pipeline(memory=None,
                                steps=[('kneighborsregressor',
                                        KNeighborsRegressor(algorithm='auto',
                                                            leaf_size=30,
                                                            metric='minkowski',
                                                            metric_params=None,
                                                            n_jobs=None,
                                                            n_neighbors=5, p=2,
                                                            weights='uniform'))],
                                verbose=False),
             iid='deprecated', n_jobs=-1,
             param_grid={'kneighborsregr...
                                                            'brute'],
                         'kneighborsregressor__n_neighbors': array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]),
                         'kneighborsregressor__weights': ['uniform',
                                                          'distance']},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring='neg_mean_absolute_error', verbose=0)

Enter to Rename, ⇧Enter to Preview

{'kneighborsregressor__algorithm': 'ball_tree',
 'kneighborsregressor__n_neighbors': 26,
 'kneighborsregressor__weights': 'distance'}

Enter to Rename, ⇧Enter to Preview

Test

y_predicted_train = knn_model.predict(x_train)
y_predicted_test  = knn_model.predict(x_test)
 
_, reg_test_scores['knn'] = reg_scores_compare({'train': y_predicted_train, 
                                                'test': y_predicted_test},
                                               {'train': y_train, 
                                                'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │              0 │       8.46876 │
├──────────┼────────────────┼───────────────┤
│ rmse     │              0 │       2.91011 │
├──────────┼────────────────┼───────────────┤
│ mae      │              0 │       2.01901 │
├──────────┼────────────────┼───────────────┤
│ medae    │              0 │       1.42325 │
├──────────┼────────────────┼───────────────┤
│ R2       │              1 │       0.33114 │
╘══════════╧════════════════╧═══════════════╛

Так как KNN просто "запоминает" train выборку, то и метрики на ней наилучшие.
Однако на отложенных данных видим, что обобщающая способность далеко не лучшая

No G1

Enter to Rename, ⇧Enter to Preview

CPU times: user 2.04 s, sys: 70.9 ms, total: 2.11 s
Wall time: 12.5 s

GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),
             error_score=nan,
             estimator=Pipeline(memory=None,
                                steps=[('kneighborsregressor',
                                        KNeighborsRegressor(algorithm='auto',
                                                            leaf_size=30,
                                                            metric='minkowski',
                                                            metric_params=None,
                                                            n_jobs=None,
                                                            n_neighbors=5, p=2,
                                                            weights='uniform'))],
                                verbose=False),
             iid='deprecated', n_jobs=-1,
             param_grid={'kneighborsregr...
                                                            'brute'],
                         'kneighborsregressor__n_neighbors': array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16, 17,
       18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34,
       35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49]),
                         'kneighborsregressor__weights': ['uniform',
                                                          'distance']},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring='neg_mean_absolute_error', verbose=0)

Enter to Rename, ⇧Enter to Preview

{'kneighborsregressor__algorithm': 'ball_tree',
 'kneighborsregressor__n_neighbors': 26,
 'kneighborsregressor__weights': 'distance'}

Enter to Rename, ⇧Enter to Preview

Test

y_predicted_train = knn_model_nog1.predict(x_no_g1_train)
y_predicted_test  = knn_model_nog1.predict(x_no_g1_test)
 
_, reg_test_scores_nog1['knn'] = reg_scores_compare({'train': y_predicted_train, 
                                                     'test': y_predicted_test},
                                                    {'train': y_train, 
                                                     'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │              0 │     10.6234   │
├──────────┼────────────────┼───────────────┤
│ rmse     │              0 │      3.25935  │
├──────────┼────────────────┼───────────────┤
│ mae      │              0 │      2.32431  │
├──────────┼────────────────┼───────────────┤
│ medae    │              0 │      1.6747   │
├──────────┼────────────────┼───────────────┤
│ R2       │              1 │      0.160968 │
╘══════════╧════════════════╧═══════════════╛

Без G1 параметра ожидаемо хуже
Также можно заметить, что grid.best_params_ такие же как и у моделей, обучаемых на выборке со всеми признаками

Random Forest

Enter to Rename, ⇧Enter to Preview

Код

Текст

params = {
    'randomforestregressor__n_estimators': np.arange(150, 250, 1), 
    # до этого сделал выборку 1 - 250 c шагом 5
    # оценил лучший подобранный параметр
    # и сейчас сузил выборку для подбора лушчего значения
    'randomforestregressor__max_features': ['auto', 'sqrt', 'log2']
}

Код

Текст

Enter to Rename, ⇧Enter to Preview

CPU times: user 11.8 s, sys: 766 ms, total: 12.6 s
Wall time: 10min 40s

GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),
             error_score=nan,
             estimator=Pipeline(memory=None,
                                steps=[('randomforestregressor',
                                        RandomForestRegressor(bootstrap=True,
                                                              ccp_alpha=0.0,
                                                              criterion='mse',
                                                              max_depth=None,
                                                              max_features='auto',
                                                              max_leaf_nodes=None,
                                                              max_samples=None,
                                                              min_impurity_decrease=0.0,
                                                              min_impurity_split=None,
                                                              min_samples_leaf=...
       189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201,
       202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214,
       215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227,
       228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240,
       241, 242, 243, 244, 245, 246, 247, 248, 249])},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring='neg_mean_absolute_error', verbose=0)

Код

Текст

Enter to Rename, ⇧Enter to Preview

{'randomforestregressor__max_features': 'auto',
 'randomforestregressor__n_estimators': 218}

Enter to Rename, ⇧Enter to Preview

y_predicted_train = rf_model.predict(x_train)
y_predicted_test  = rf_model.predict(x_test)
 
_, reg_test_scores['random forest'] = reg_scores_compare({'train': y_predicted_train, 
                                                          'test': y_predicted_test},
                                                         {'train': y_train, 
                                                          'test': y_test})

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │       0.597778 │      3.64406  │
├──────────┼────────────────┼───────────────┤
│ rmse     │       0.773161 │      1.90894  │
├──────────┼────────────────┼───────────────┤
│ mae      │       0.528149 │      1.38912  │
├──────────┼────────────────┼───────────────┤
│ medae    │       0.380734 │      1.00459  │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.961189 │      0.712193 │
╘══════════╧════════════════╧═══════════════╛

Мы видим довольно сильное падение в качестве модели на test выборке
Можно поробовать избавиться от этого подобрав max_depth деревьев

Enter to Rename, ⇧Enter to Preview

GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),
             error_score=nan,
             estimator=Pipeline(memory=None,
                                steps=[('randomforestregressor',
                                        RandomForestRegressor(bootstrap=True,
                                                              ccp_alpha=0.0,
                                                              criterion='mse',
                                                              max_depth=None,
                                                              max_features='auto',
                                                              max_leaf_nodes=None,
                                                              max_samples=None,
                                                              min_impurity_decrease=0.0,
                                                              min_impurity_split=None,
                                                              min_samples_leaf=...
                                                              n_estimators=218,
                                                              n_jobs=None,
                                                              oob_score=False,
                                                              random_state=None,
                                                              verbose=0,
                                                              warm_start=False))],
                                verbose=False),
             iid='deprecated', n_jobs=-1,
             param_grid={'randomforestregressor__max_depth': array([ 2,  4,  6,  8, 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, 30, 32, 34,
       36, 38, 40, 42, 44, 46, 48])},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring='neg_mean_absolute_error', verbose=0)

Enter to Rename, ⇧Enter to Preview

{'randomforestregressor__max_depth': 6}

Enter to Rename, ⇧Enter to Preview

y_predicted_train = rf_model.predict(x_train)
y_predicted_test  = rf_model.predict(x_test)
 
_, reg_test_scores['random forest'] = reg_scores_compare({'train': y_predicted_train, 
                                                          'test': y_predicted_test},
                                                         {'train': y_train, 
                                                          'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │       1.8317   │      3.59326  │
├──────────┼────────────────┼───────────────┤
│ rmse     │       1.3534   │      1.89559  │
├──────────┼────────────────┼───────────────┤
│ mae      │       1.00602  │      1.38352  │
├──────────┼────────────────┼───────────────┤
│ medae    │       0.776568 │      1.05151  │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.881076 │      0.716205 │
╘══════════╧════════════════╧═══════════════╛

Получилось немного улучшить метрики

No G1

Enter to Rename, ⇧Enter to Preview

CPU times: user 6.3 s, sys: 454 ms, total: 6.75 s
Wall time: 5min 26s

GridSearchCV(cv=KFold(n_splits=5, random_state=None, shuffle=True),
             error_score=nan,
             estimator=Pipeline(memory=None,
                                steps=[('randomforestregressor',
                                        RandomForestRegressor(bootstrap=True,
                                                              ccp_alpha=0.0,
                                                              criterion='mse',
                                                              max_depth=None,
                                                              max_features='auto',
                                                              max_leaf_nodes=None,
                                                              max_samples=None,
                                                              min_impurity_decrease=0.0,
                                                              min_impurity_split=None,
                                                              min_samples_leaf=...
                         'randomforestregressor__n_estimators': array([150, 152, 154, 156, 158, 160, 162, 164, 166, 168, 170, 172, 174,
       176, 178, 180, 182, 184, 186, 188, 190, 192, 194, 196, 198, 200,
       202, 204, 206, 208, 210, 212, 214, 216, 218, 220, 222, 224, 226,
       228, 230, 232, 234, 236, 238, 240, 242, 244, 246, 248])},
             pre_dispatch='2*n_jobs', refit=True, return_train_score=False,
             scoring='neg_mean_absolute_error', verbose=0)

Enter to Rename, ⇧Enter to Preview

{'randomforestregressor__max_features': 'auto',
 'randomforestregressor__n_estimators': 176}

Enter to Rename, ⇧Enter to Preview

Test

y_predicted_train = rf_model_nog1.predict(x_no_g1_train)
y_predicted_test  = rf_model_nog1.predict(x_no_g1_test)
 
_, reg_test_scores_nog1['random forest'] = \
                            reg_scores_compare({'train': y_predicted_train, 
                                                'test': y_predicted_test},
                                               {'train': y_train, 
                                                'test': y_test})

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │       1.33962  │     10.6123   │
├──────────┼────────────────┼───────────────┤
│ rmse     │       1.15742  │      3.25765  │
├──────────┼────────────────┼───────────────┤
│ mae      │       0.8408   │      2.32433  │
├──────────┼────────────────┼───────────────┤
│ medae    │       0.602273 │      1.60795  │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.913025 │      0.161843 │
╘══════════╧════════════════╧═══════════════╛

RandomForestRegressor, обученный на данных без признака G1 имеет плохую обобщающую способность

Enter to Rename, ⇧Enter to Preview

{'randomforestregressor__max_depth': 32}

Enter to Rename, ⇧Enter to Preview

Test

Enter to Rename, ⇧Enter to Preview

SCORES:
╒══════════╤════════════════╤═══════════════╕
│ metric   │   train scores │   test scores │
╞══════════╪════════════════╪═══════════════╡
│ mse      │       1.35514  │     10.7822   │
├──────────┼────────────────┼───────────────┤
│ rmse     │       1.1641   │      3.28363  │
├──────────┼────────────────┼───────────────┤
│ mae      │       0.842246 │      2.34461  │
├──────────┼────────────────┼───────────────┤
│ medae    │       0.614679 │      1.64679  │
├──────────┼────────────────┼───────────────┤
│ R2       │       0.912017 │      0.148425 │
╘══════════╧════════════════╧═══════════════╛

Улучшить результаты не получилось

Results

Код

Текст

def compare_estimators(scores: dict, printable=True):
  estimators_names = list(scores.keys())
  headers = ['estimators'] + list(scores[estimators_names[0]].keys())
  table = []
 
  for estimator in estimators_names:
    row = [estimator]
    for metric_key in scores[estimator].keys():
      row.append(scores[estimator][metric_key])
    table.append(row)
 
  # print
  print(tb(table, headers=headers, tablefmt='fancy_grid'))
 
  return table

with G1

Enter to Rename, ⇧Enter to Preview

{'R2': 0.7162051857150782,
 'mae': 1.3835179330702032,
 'medae': 1.0515141031025337,
 'mse': 3.5932616642774806,
 'rmse': 1.895590057021159}

Enter to Rename, ⇧Enter to Preview

╒═══════════════╤══════════╤═════════╤═════════╤═════════╤═══════════╕
│ estimators    │      mse │    rmse │     mae │   medae │        R2 │
╞═══════════════╪══════════╪═════════╪═════════╪═════════╪═══════════╡
│ losso         │ 11.5461  │ 3.39796 │ 2.41408 │ 1.70175 │ 0.0880885 │
├───────────────┼──────────┼─────────┼─────────┼─────────┼───────────┤
│ ridge         │  4.91063 │ 2.21599 │ 1.59266 │ 1.27257 │ 0.61216   │
├───────────────┼──────────┼─────────┼─────────┼─────────┼───────────┤
│ poly          │  5.02588 │ 2.24185 │ 1.62587 │ 1.2604  │ 0.603057  │
├───────────────┼──────────┼─────────┼─────────┼─────────┼───────────┤
│ knn           │  8.46876 │ 2.91011 │ 2.01901 │ 1.42325 │ 0.33114   │
├───────────────┼──────────┼─────────┼─────────┼─────────┼───────────┤
│ random forest │  3.59326 │ 1.89559 │ 1.38352 │ 1.05151 │ 0.716205  │
╘═══════════════╧══════════╧═════════╧═════════╧═════════╧═══════════╛

Код

Текст

Best Regression (G1)

model: RandomForestRegressor
params: n_estimators=218, max_depth=6
score (R2): 0.716

Код

Текст

no G1

╒═══════════════╤═════════╤═════════╤═════════╤═════════╤════════════╕
│ estimators    │     mse │    rmse │     mae │   medae │         R2 │
╞═══════════════╪═════════╪═════════╪═════════╪═════════╪════════════╡
│ losso         │ 13.0908 │ 3.61813 │ 2.61368 │ 1.78922 │ -0.0339102 │
├───────────────┼─────────┼─────────┼─────────┼─────────┼────────────┤
│ ridge         │ 11.0084 │ 3.3179  │ 2.41881 │ 1.87332 │  0.130556  │
├───────────────┼─────────┼─────────┼─────────┼─────────┼────────────┤
│ poly          │ 13.0745 │ 3.61587 │ 2.66703 │ 2.05421 │ -0.0326215 │
├───────────────┼─────────┼─────────┼─────────┼─────────┼────────────┤
│ knn           │ 10.6234 │ 3.25935 │ 2.32431 │ 1.6747  │  0.160968  │
├───────────────┼─────────┼─────────┼─────────┼─────────┼────────────┤
│ random forest │ 10.8576 │ 3.29509 │ 2.35762 │ 1.62844 │  0.142469  │
╘═══════════════╧═════════╧═════════╧═════════╧═════════╧════════════╛

Код

Текст

Best Regression (no G1)

model: KNeighborsRegressor
params: n_neighbors=26, algorithm="ball_tree", weights="distance"
score (R2): 0.16

Код

Текст

Часть 3. Бинарная классификация

Код

Текст

Решите задачу бинарной классификации: постройте модель, предсказывающую, сдаст студент предмет (G3 >= 8) или не сдаст (G3 < 8).
При решении задачи нельзя использовать признаки G1 и G2.

Код

Текст

Уберем из использования G1

Код

Текст

Код

Текст

Создадим новый признак для предсказания

Код

Текст

860    1
269    1
581    0
775    1
669    1
      ..
866    0
742    1
74     0
176    1
338    1
Name: G3, Length: 835, dtype: int64

Код

Текст

Проверим распредление

Код

Текст

Train:
1    0.894611
0    0.105389
Name: G3, dtype: float64
Test:
1    0.942584
0    0.057416
Name: G3, dtype: float64

Код

Текст

Код

Текст

Распределение практически одинаковое и не будет негавтино влиять на процесс обучения

Код

Текст

Код

Текст

Задание 1

Постройте дерево решений глубины 5 (остальные параметры по умолчанию), оцените качество на 5-fold валидации.
Для одного из деревьев (т.е. обученного на одной из итераций кросс-валидации) выведите само дерево - постройте график или выведите в текстовом виде. По структуре дерева сформулируйте правила, по которым принимается решение.
Сравните между собой деревья решений, полученных на различных итерациях 5-fold валидации. Сделайте вывод, насколько сильно они похожи или различаются между собой.

Код

Текст

5-fold validation

Код

Текст

five_fold = StratifiedKFold(n_splits=5, shuffle=True)
    train_x = x_train[train_index]
    train_y = y_binary_train.iloc[train_index]
    # model
    model = DecisionTreeClassifier(max_depth=5)
    model.fit(train_x, train_y)
    test_x = x_train[test_index]
    test_y = y_binary_train.iloc[test_index]
    # cv train 
    # cv test
    # score
    tree_models.append(model)
    # pred_y = model.predict(test_x)
    score = model.score(test_x, test_y)
    tree_scores.append(score)
 
for train_index, test_index in five_fold.split(x_train, y_train):
tree_models = []
tree_scores = []
print("CV MEAN SCORE: ", np.mean(tree_scores))
 
print("All scores: ", ', '.join(list(map(lambda x: "{:.3f}".format(x), tree_scores))))

CV MEAN SCORE:  0.8802395209580839
All scores:  0.862, 0.874, 0.868, 0.904, 0.892

Код

Текст

Visualize

Enter to Rename, ⇧Enter to Preview

def export_graphviz(decision_tree, out_file=None, max_depth=None, feature_names=None, class_names=None, label='all', filled=False, leaves_parallel=False, impurity=True, node_ids=False, proportion=False, rotate=False, rounded=False, special_characters=False, precision=3)

Открыть в новой вкладке Посмотреть исходный код

Export a decision tree in DOT format.

This function generates a GraphViz representation of the decision tree, which is then written into out_file. Once exported, graphical renderings can be generated using, for example:

$ dot -Tps tree.dot -o tree.ps      (PostScript format)
$ dot -Tpng tree.dot -o tree.png    (PNG format)

The sample counts that are shown are weighted with any sample_weights that might be present.

Parameters

decision_tree : decision tree classifier
The decision tree to be exported to GraphViz.

out_file : file object or string, optional (default=None)
Handle or name of the output file. If None, the result is returned as a string.

max_depth : int, optional (default=None)
The maximum depth of the representation. If None, the tree is fully generated.

feature_names : list of strings, optional (default=None)
Names of each of the features.

class_names : list of strings, bool or None, optional (default=None)
Names of each of the target classes in ascending numerical order. Only relevant for classification and not supported for multi-output. If True, shows a symbolic representation of the class name.

label : {'all', 'root', 'none'}, optional (default='all')
Whether to show informative labels for impurity, etc. Options include 'all' to show at every node, 'root' to show only at the top root node, or 'none' to not show at any node.

filled : bool, optional (default=False)
When set to True, paint nodes to indicate majority class for classification, extremity of values for regression, or purity of node for multi-output.

leaves_parallel : bool, optional (default=False)
When set to True, draw all leaf nodes at the bottom of the tree.

impurity : bool, optional (default=True)
When set to True, show the impurity at each node.

node_ids : bool, optional (default=False)
When set to True, show the ID number on each node.

proportion : bool, optional (default=False)
When set to True, change the display of 'values' and/or 'samples' to be proportions and percentages respectively.

rotate : bool, optional (default=False)
When set to True, orient tree left to right rather than top-down.

rounded : bool, optional (default=False)
When set to True, draw node boxes with rounded corners and use Helvetica fonts instead of Times-Roman.

special_characters : bool, optional (default=False)
When set to False, ignore special characters for PostScript compatibility.

precision : int, optional (default=3)
Number of digits of precision for floating point in the values of impurity, threshold and value attributes of each node.

Returns

dot_data : string
String representation of the input tree in GraphViz dot format. Only returned if out_file is None.

Examples

>>> from sklearn.datasets import load_iris
>>> from sklearn import tree

>>> clf = tree.DecisionTreeClassifier()
>>> iris = load_iris()

>>> clf = clf.fit(iris.data, iris.target)
>>> tree.export_graphviz(clf)
'digraph Tree {...

Код

Текст

dot_data = export_graphviz(tree_models[0], 
                           out_file=None, 
                           feature_names=learning_data.drop(columns=['G1']).columns, 
                           class_names=['failed','passed'],
                           filled=True, 
                           rounded=True, 
                           special_characters=True)
 
graph = graphviz.Source(dot_data) 
graph.render("decision_tree_example.pdf") 

Код

Текст

Код

Текст

Compare trees

Код

Текст

Задание 2

На кросс-валидации (5-fold из 2 повторений) оцените, как меняется качество модели Random Forest с ростом числа деревьев (при дефолтных значениях остальных параметров). Провизуализируйте результаты. Сколько деревьев достаточно в данном случае и почему?
NB: В сравнение включите конфигурацию, аналогичную простому дереву решений.

Код

Текст

Код

Текст

def bi_class_scores(predicted, labels, printable=False):
  scores = dict()
  scores['precision'] = metrics.precision_score(labels, predicted)
  scores['recall'] = metrics.recall_score(labels, predicted)                                                
  scores['f1'] = metrics.f1_score(labels, predicted)
  scores['accuracy'] = metrics.accuracy_score(labels, predicted)
  # print
  if printable:
    scores_table = [(key, value) for key, value in scores.items()]
    print('SCORES:')
    print(tb(scores_table, headers=('metric', 'score'), tablefmt="fancy_grid"))
  scores['roc-auc'] = metrics.roc_auc_score(labels, predicted)
  return scores

Код

Текст

Код

Текст

Код

Текст

# your code here
 
double_five_fold = RepeatedStratifiedKFold(n_splits=5, n_repeats=2)
table_rows = []
 
  for train_index, test_index in five_fold.split(x_train, y_binary_test):
      # cv train 
      train_x = x_train[train_index]
      train_y = y_binary_train.iloc[train_index]
      # cv test
      test_x = x_train[test_index]
      test_y = y_binary_train.iloc[test_index]
      # model
      model = RandomForestClassifier(n_estimators=trees_number)
      model.fit(train_x, train_y)
      # score
      pred_y = model.predict(test_x)
      scores.append(bi_class_scores(pred_y, test_y))
trees_number_df = pd.DataFrame(table_rows, 
for trees_number in trees_test_numbers:
# trees_test_numbers = [1, 5, 10, 15, 20, 40, 75, 100, 125, 150, 175, 200]
 
  scores = []
  for example in scores:
  mean_scores = {key: [] for key in scores[0].keys()}
 
    for key, value in example.items():
      mean_scores[key].append(value)
  
  for key in mean_scores.keys():
    mean_scores[key] = np.mean(mean_scores[key])
  
    table_rows.append(row)
    row = (trees_number, mean_scores[key], key)
    # trees, score, metric
                               columns=['trees_number', 'score', 'metric'])
trees_test_numbers = np.arange(1, 50, 1)
trees_number_df

Enter to Rename, ⇧Enter to Preview

1/1

np.arange

arange([start,] stop[, step,], dtype=None) Return evenly spaced values within a given interval. Values are generated within the half-open interval ``[start, stop)`` (in other words, the interval including `start` but excluding `stop`). For integer arguments the function is equivalent to the Python built-in `range` function, but returns an ndarray rather than a list. When using a non-integer step, such as 0.1, the results will often not be consistent. It is better to use `numpy.linspace` for these cases. Parameters ---------- start : number, optional Start of interval. The interval includes this value. The default start value is 0. stop : number End of interval. The interval does not include this value, except in some cases where `step` is not an integer and floating point round-off affects the length of `out`. step : number, optional Spacing between values. For any output `out`, this is the distance between two adjacent values, ``out[i+1] - out[i]``. The default step size is 1. If `step` is specified as a position argument, `start` must also be given. dtype : dtype The type of the output array. If `dtype` is not given, infer the data type from the other input arguments. Returns ------- arange : ndarray Array of evenly spaced values. For floating point arguments, the length of the result is ``ceil((stop - start)/step)``. Because of floating point overflow, this rule may result in the last element of `out` being greater than `stop`. See Also -------- numpy.linspace : Evenly spaced numbers with careful handling of endpoints. numpy.ogrid: Arrays of evenly spaced numbers in N-dimensions. numpy.mgrid: Grid-shaped arrays of evenly spaced numbers in N-dimensions. Examples -------- >>> np.arange(3) array([0, 1, 2]) >>> np.arange(3.0) array([ 0., 1., 2.]) >>> np.arange(3,7) array([3, 4, 5, 6]) >>> np.arange(3,7,2) array([3, 5])

Код

Текст

Код

Текст

Вывод:
Основываясь на полученном графике оптимальное количество деревьев находится в районе 7-10

Код

Текст

Задание 3

Настройте гиперпараметры модели Random Forest на 5-fold валдиации. В качестве метрики используйте F1-score. Замерьте время, затраченное на вычисления.
Обучите Random Forest с настроенными параметрами на всех данных для моделирования. На отложенной выборке оцените качество (F1-score) всего ансамбля и каждого дерева отдельно. Постройте график распределения качества деревьев в ансамбле и сравните результаты с качеством всего леса. Дайте комментарий.
Выведите важность признаков в Random Forest, сделайте выводы.

Код

Текст

Enter to Rename, ⇧Enter to Preview

def cross_val_score(estimator, X, y=None, groups=None, scoring=None, cv=None, n_jobs=None, verbose=0, fit_params=None, pre_dispatch='2*n_jobs', error_score=np.nan)

Открыть в новой вкладке Посмотреть исходный код

Evaluate a score by cross-validation

Read more in the User Guide <cross_validation>.

Parameters

estimator : estimator object implementing 'fit'
The object to use to fit the data.

X : array-like
The data to fit. Can be for example a list, or an array.

y : array-like, optional, default: None
The target variable to try to predict in the case of supervised learning.

groups : array-like, with shape (n_samples,), optional
Group labels for the samples used while splitting the dataset into train/test set. Only used in conjunction with a "Group" cv instance (e.g., GroupKFold).

scoring : string, callable or None, optional, default: None
A string (see model evaluation documentation) or a scorer callable object / function with signature scorer(estimator, X, y) which should return only a single value.

Similar to cross_validate but only a single metric is permitted.

If None, the estimator's default scorer (if available) is used.

cv : int, cross-validation generator or an iterable, optional
Determines the cross-validation splitting strategy. Possible inputs for cv are:

None, to use the default 5-fold cross validation,
integer, to specify the number of folds in a (Stratified)KFold,
CV splitter,
An iterable yielding (train, test) splits as arrays of indices.

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

Refer User Guide <cross_validation> for the various cross-validation strategies that can be used here.

n_jobs : int or None, optional (default=None)
The number of CPUs to use to do the computation. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary <n_jobs> for more details.

verbose : integer, optional
The verbosity level.

fit_params : dict, optional
Parameters to pass to the fit method of the estimator.

pre_dispatch : int, or string, optional
Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be:

- None, in which case all the jobs are immediately

created and spawned. Use this for lightweight and fast-running jobs, to avoid delays due to on-demand spawning of the jobs

- An int, giving the exact number of total jobs that are

spawned

- A string, giving an expression as a function of n\_jobs,

as in '2*n_jobs'

error_score : 'raise' or numeric
Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error.

Returns

scores : array of float, shape=(len(list(cv)),)
Array of scores of the estimator for each run of the cross validation.

Examples

>>> from sklearn import datasets, linear_model
>>> from sklearn.model_selection import cross_val_score
>>> diabetes = datasets.load_diabetes()
>>> X = diabetes.data[:150]
>>> y = diabetes.target[:150]
>>> lasso = linear_model.Lasso()
>>> print(cross_val_score(lasso, X, y, cv=3))
[0.33150734 0.08022311 0.03531764]

Parameters

n_estimators : integer, optional (default=100)
The number of trees in the forest.

criterion : string, optional (default="gini")
The function to measure the quality of a split. Supported criteria are "gini" for the Gini impurity and "entropy" for the information gain.
Note: this parameter is tree-specific.

max_depth : integer or None, optional (default=None)
The maximum depth of the tree. If None, then nodes are expanded until all leaves are pure or until all leaves contain less than min_samples_split samples.

min_samples_split : int, float, optional (default=2)
The minimum number of samples required to split an internal node:

If int, then consider min_samples_split as the minimum number.
If float, then min_samples_split is a fraction and
ceil(min_samples_split * n_samples) are the minimum number of samples for each split.

min_samples_leaf : int, float, optional (default=1)
The minimum number of samples required to be at a leaf node. A split point at any depth will only be considered if it leaves at least min_samples_leaf training samples in each of the left and right branches. This may have the effect of smoothing the model, especially in regression.

If int, then consider min_samples_leaf as the minimum number.
If float, then min_samples_leaf is a fraction and
ceil(min_samples_leaf * n_samples) are the minimum number of samples for each node.

min_weight_fraction_leaf : float, optional (default=0.)
The minimum weighted fraction of the sum total of weights (of all the input samples) required to be at a leaf node. Samples have equal weight when sample_weight is not provided.

max_features : int, float, string or None, optional (default="auto")
The number of features to consider when looking for the best split:

If int, then consider max_features features at each split.
If float, then max_features is a fraction and
int(max_features * n_features) features are considered at each split.
If "auto", then max_features=sqrt(n_features).
If "sqrt", then max_features=sqrt(n_features) (same as "auto").
If "log2", then max_features=log2(n_features).
If None, then max_features=n_features.

Note: the search for a split does not stop until at least one valid partition of the node samples is found, even if it requires to effectively inspect more than max_features features.

max_leaf_nodes : int or None, optional (default=None)
Grow trees with max_leaf_nodes in best-first fashion. Best nodes are defined as relative reduction in impurity. If None then unlimited number of leaf nodes.

min_impurity_decrease : float, optional (default=0.)
A node will be split if this split induces a decrease of the impurity greater than or equal to this value.

The weighted impurity decrease equation is the following:

N_t / N * (impurity - N_t_R / N_t * right_impurity
                    - N_t_L / N_t * left_impurity)

where N is the total number of samples, N_t is the number of samples at the current node, N_t_L is the number of samples in the left child, and N_t_R is the number of samples in the right child.

N, N_t, N_t_R and N_t_L all refer to the weighted sum, if sample_weight is passed.

min_impurity_split : float, (default=1e-7)
Threshold for early stopping in tree growth. A node will split if its impurity is above the threshold, otherwise it is a leaf.

bootstrap : boolean, optional (default=True)
Whether bootstrap samples are used when building trees. If False, the whole datset is used to build each tree.

oob_score : bool (default=False)
Whether to use out-of-bag samples to estimate the generalization accuracy.

n_jobs : int or None, optional (default=None)
The number of jobs to run in parallel. fit, predict, decision_path and apply are all parallelized over the trees. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary <n_jobs> for more details.

random_state : int, RandomState instance or None, optional (default=None)
Controls both the randomness of the bootstrapping of the samples used when building trees (if bootstrap=True) and the sampling of the features to consider when looking for the best split at each node (if max_features < n_features). See Glossary <random_state> for details.

verbose : int, optional (default=0)
Controls the verbosity when fitting and predicting.

warm_start : bool, optional (default=False)
When set to True, reuse the solution of the previous call to fit and add more estimators to the ensemble, otherwise, just fit a whole new forest. See the Glossary <warm_start>.

class_weight : dict, list of dicts, "balanced", "balanced_subsample" or None, optional (default=None)
Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one. For multi-output problems, a list of dicts can be provided in the same order as the columns of y.

Note that for multioutput (including multilabel) weights should be defined for each class of every column in its own dict. For example, for four-class multilabel classification weights should be [{0: 1, 1: 1}, {0: 1, 1: 5}, {0: 1, 1: 1}, {0: 1, 1: 1}] instead of [{1:1}, {2:5}, {3:1}, {4:1}].

The "balanced" mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y))

The "balanced_subsample" mode is the same as "balanced" except that weights are computed based on the bootstrap sample for every tree grown.

For multi-output, the weights of each column of y will be multiplied.

Note that these weights will be multiplied with sample_weight (passed through the fit method) if sample_weight is specified.

ccp_alpha : non-negative float, optional (default=0.0)
Complexity parameter used for Minimal Cost-Complexity Pruning. The subtree with the largest cost complexity that is smaller than ccp_alpha will be chosen. By default, no pruning is performed. See minimal_cost_complexity_pruning for details.

max_samples : int or float, default=None
If bootstrap is True, the number of samples to draw from X to train each base estimator.

If None (default), then draw X.shape[0] samples.
If int, then draw max_samples samples.
If float, then draw max_samples * X.shape[0] samples. Thus, max_samples should be in the interval (0, 1).

Attributes

base_estimator_ : DecisionTreeClassifier
The child estimator template used to create the collection of fitted sub-estimators.

estimators_ : list of DecisionTreeClassifier
The collection of fitted sub-estimators.

classes_ : array of shape (n_classes,) or a list of such arrays
The classes labels (single output problem), or a list of arrays of class labels (multi-output problem).

n_classes_ : int or list
The number of classes (single output problem), or a list containing the number of classes for each output (multi-output problem).

n_features_ : int
The number of features when fit is performed.

n_outputs_ : int
The number of outputs when fit is performed.

feature_importances_ : ndarray of shape (n_features,)
The feature importances (the higher, the more important the feature).

oob_score_ : float
Score of the training dataset obtained using an out-of-bag estimate. This attribute exists only when oob_score is True.

oob_decision_function_ : array of shape (n_samples, n_classes)
Decision function computed with out-of-bag estimate on the training set. If n_estimators is small it might be possible that a data point was never left out during the bootstrap. In this case, oob_decision_function_ might contain NaN. This attribute exists only when oob_score is True.

Examples

>>> from sklearn.ensemble import RandomForestClassifier
>>> from sklearn.datasets import make_classification

>>> X, y = make_classification(n_samples=1000, n_features=4,
...                            n_informative=2, n_redundant=0,
...                            random_state=0, shuffle=False)
>>> clf = RandomForestClassifier(max_depth=2, random_state=0)
>>> clf.fit(X, y)
RandomForestClassifier(max_depth=2, random_state=0)
>>> print(clf.feature_importances_)
[0.14205973 0.76664038 0.0282433  0.06305659]
>>> print(clf.predict([[0, 0, 0, 0]]))
[1]

Notes

The default values for the parameters controlling the size of the trees (e.g. max_depth, min_samples_leaf, etc.) lead to fully grown and unpruned trees which can potentially be very large on some data sets. To reduce memory consumption, the complexity and size of the trees should be controlled by setting those parameter values.

The features are always randomly permuted at each split. Therefore, the best found split may vary, even with the same training data, max_features=n_features and bootstrap=False, if the improvement of the criterion is identical for several splits enumerated during the search of the best split. To obtain a deterministic behaviour during fitting, random_state has to be fixed.

References

Parameters

X : iterable
Training data. Must fulfill input requirements of first step of the pipeline.

y : iterable, default=None
Training targets. Must fulfill label requirements for all steps of the pipeline.

**fit_params : dict of string -> object
Parameters passed to the fit method of each step, where each parameter name is prefixed such that parameter p for step s has key s__p.

Returns

self : Pipeline
This estimator

RandomForestClassifier(bootstrap=True, ccp_alpha=0.0, class_weight=None,
                       criterion='entropy', max_depth=None, max_features='auto',
                       max_leaf_nodes=None, max_samples=None,
                       min_impurity_decrease=0.0, min_impurity_split=None,
                       min_samples_leaf=1, min_samples_split=2,
                       min_weight_fraction_leaf=0.0, n_estimators=20,
                       n_jobs=None, oob_score=False, random_state=None,
                       verbose=0, warm_start=False)

Код

Текст

Enter to Rename, ⇧Enter to Preview

classification_report(y_true, y_pred, labels=None, target_names=None, sample_weight=None, digits=2, output_dict=False, zero_division='warn')

Открыть в новой вкладке Посмотреть исходный код

Build a text report showing the main classification metrics

Read more in the User Guide <classification_report>.

Parameters

y_true : 1d array-like, or label indicator array / sparse matrix
Ground truth (correct) target values.

y_pred : 1d array-like, or label indicator array / sparse matrix
Estimated targets as returned by a classifier.

labels : array, shape = [n_labels]
Optional list of label indices to include in the report.

target_names : list of strings
Optional display names matching the labels (same order).

sample_weight : array-like of shape (n_samples,), default=None
Sample weights.

digits : int
Number of digits for formatting output floating point values. When output_dict is True, this will be ignored and the returned values will not be rounded.

output_dict : bool (default = False)
If True, return output as dict

zero_division : "warn", 0 or 1, default="warn"
Sets the value to return when there is a zero division. If set to "warn", this acts as 0, but warnings are also raised.

Returns

report : string / dict
Text summary of the precision, recall, F1 score for each class. Dictionary returned if output_dict is True. Dictionary has the following structure:

{'label 1': {'precision':0.5,
             'recall':1.0,
             'f1-score':0.67,
             'support':1},
 'label 2': { ... },
  ...
}

The reported averages include macro average (averaging the unweighted mean per label), weighted average (averaging the support-weighted mean per label), and sample average (only for multilabel classification). Micro average (averaging the total true positives, false negatives and false positives) is only shown for multi-label or multi-class with a subset of classes, because it corresponds to accuracy otherwise. See also precision_recall_fscore_support for more details on averages.

Note that in binary classification, recall of the positive class is also known as "sensitivity"; recall of the negative class is "specificity".

Examples

>>> from sklearn.metrics import classification_report
>>> y_true = [0, 1, 2, 2, 2]
>>> y_pred = [0, 0, 2, 2, 1]
>>> target_names = ['class 0', 'class 1', 'class 2']
>>> print(classification_report(y_true, y_pred, target_names=target_names))
              precision    recall  f1-score   support
<BLANKLINE>
     class 0       0.50      1.00      0.67         1
     class 1       0.00      0.00      0.00         1
     class 2       1.00      0.67      0.80         3
<BLANKLINE>
    accuracy                           0.60         5
   macro avg       0.50      0.56      0.49         5
weighted avg       0.70      0.60      0.61         5
<BLANKLINE>
>>> y_pred = [1, 1, 0]
>>> y_true = [1, 1, 1]
>>> print(classification_report(y_true, y_pred, labels=[1, 2, 3]))
              precision    recall  f1-score   support
<BLANKLINE>
           1       1.00      0.67      0.80         3
           2       0.00      0.00      0.00         0
           3       0.00      0.00      0.00         0
<BLANKLINE>
   micro avg       1.00      0.67      0.80         3
   macro avg       0.33      0.22      0.27         3
weighted avg       1.00      0.67      0.80         3
<BLANKLINE>

F1:  0.9752475247524752
              precision    recall  f1-score   support

           0       1.00      0.17      0.29        12
           1       0.95      1.00      0.98       197

    accuracy                           0.95       209
   macro avg       0.98      0.58      0.63       209
weighted avg       0.95      0.95      0.94       209

Код

Текст

0     0.918206
1     0.901333
2     0.938144
3     0.930591
4     0.936061
5     0.941176
6     0.929870
7     0.923483
8     0.934726
9     0.948980
10    0.932292
11    0.920213
12    0.940568
13    0.927461
14    0.938462
15    0.915789
16    0.927083
17    0.924282
18    0.927083
19    0.918635
dtype: float64

Код

Текст

Код

Текст

Вывод:
f1 score ансамбля - 0.975
f1 scores деревьев - большинство значений расположены в промежутке от 0.92-0.94

Каждое дерево по отдельности предсказывает результат хуже, чем в ансамбле.

Код

Текст

Задание 4

Примените логистическую регрессию для решения задачи, подберите оптимальные значения гиперпараметров. Оцените качество (roc auc) на 5-fold валидации из 2 повторений.
Аналогично (на такой же валидации (тех же подвыборках) с такой же метрикой) оцените качество Random Forest с подобранными в предыдущем задании параметрами. Сравните с качеством логистическом регрессии.
Обучите логистическую модель с настроенными параметрами на всех данных для моделирования. На отложенной выборке оцените качество - постройте ROC-кривую, вычислите roc auc. Вычислите аналогичную метрику для Random Forest из Задания 3, сравните точность моделей.

Код

Текст

Код

Текст

Код

Текст

Enter to Rename, ⇧Enter to Preview

1/1

def __init__(estimator, param_grid, scoring=None, n_jobs=None, iid='deprecated', refit=True, cv=None, verbose=0, pre_dispatch='2*n_jobs', error_score=np.nan, return_train_score=False)

Exhaustive search over specified parameter values for an estimator.

Important members are fit, predict.

GridSearchCV implements a "fit" and a "score" method. It also implements "predict", "predict_proba", "decision_function", "transform" and "inverse_transform" if they are implemented in the estimator used.

The parameters of the estimator used to apply these methods are optimized by cross-validated grid-search over a parameter grid.

Read more in the User Guide <grid_search>.

Parameters

estimator : estimator object.
This is assumed to implement the scikit-learn estimator interface. Either estimator needs to provide a score function, or scoring must be passed.

param_grid : dict or list of dictionaries
Dictionary with parameters names (string) as keys and lists of parameter settings to try as values, or a list of such dictionaries, in which case the grids spanned by each dictionary in the list are explored. This enables searching over any sequence of parameter settings.

scoring : string, callable, list/tuple, dict or None, default: None
A single string (see scoring_parameter) or a callable (see scoring) to evaluate the predictions on the test set.

For evaluating multiple metrics, either give a list of (unique) strings or a dict with names as keys and callables as values.

NOTE that when using custom scorers, each scorer should return a single value. Metric functions returning a list/array of values can be wrapped into multiple scorers that return one value each.

See multimetric_grid_search for an example.

If None, the estimator's score method is used.

n_jobs : int or None, optional (default=None)
Number of jobs to run in parallel. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary <n_jobs> for more details.

pre_dispatch : int, or string, optional
Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be:

- None, in which case all the jobs are immediately

created and spawned. Use this for lightweight and fast-running jobs, to avoid delays due to on-demand spawning of the jobs

- An int, giving the exact number of total jobs that are

spawned

- A string, giving an expression as a function of n\_jobs,

as in '2*n_jobs'

iid : boolean, default=False
If True, return the average score across folds, weighted by the number of samples in each test set. In this case, the data is assumed to be identically distributed across the folds, and the loss minimized is the total loss per sample, and not the mean loss across the folds.

cv : int, cross-validation generator or an iterable, optional
Determines the cross-validation splitting strategy. Possible inputs for cv are:

None, to use the default 5-fold cross validation,
integer, to specify the number of folds in a (Stratified)KFold,
CV splitter,
An iterable yielding (train, test) splits as arrays of indices.

For integer/None inputs, if the estimator is a classifier and y is either binary or multiclass, StratifiedKFold is used. In all other cases, KFold is used.

Refer User Guide <cross_validation> for the various cross-validation strategies that can be used here.

refit : boolean, string, or callable, default=True
Refit an estimator using the best found parameters on the whole dataset.

For multiple metric evaluation, this needs to be a string denoting the scorer that would be used to find the best parameters for refitting the estimator at the end.

Where there are considerations other than maximum score in choosing a best estimator, refit can be set to a function which returns the selected best_index_ given cv_results_. In that case, the best_estimator_ and best_parameters_ will be set according to the returned best_index_ while the best_score_ attribute will not be available.

The refitted estimator is made available at the best_estimator_ attribute and permits using predict directly on this GridSearchCV instance.

Also for multiple metric evaluation, the attributes best_index_, best_score_ and best_params_ will only be available if refit is set and all of them will be determined w.r.t this specific scorer.

See scoring parameter to know more about multiple metric evaluation.

verbose : integer
Controls the verbosity: the higher, the more messages.

error_score : 'raise' or numeric
Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error. Default is np.nan.

return_train_score : boolean, default=False
If False, the cv_results_ attribute will not include training scores. Computing training scores is used to get insights on how different parameter settings impact the overfitting/underfitting trade-off. However computing the scores on the training set can be computationally expensive and is not strictly required to select the parameters that yield the best generalization performance.

Examples

>>> from sklearn import svm, datasets
>>> from sklearn.model_selection import GridSearchCV
>>> iris = datasets.load_iris()
>>> parameters = {'kernel':('linear', 'rbf'), 'C':[1, 10]}
>>> svc = svm.SVC()
>>> clf = GridSearchCV(svc, parameters)
>>> clf.fit(iris.data, iris.target)
GridSearchCV(estimator=SVC(),
param_grid={'C': [1, 10], 'kernel': ('linear', 'rbf')})
>>> sorted(clf.cv_results_.keys())
['mean_fit_time', 'mean_score_time', 'mean_test_score',...
'param_C', 'param_kernel', 'params',...
'rank_test_score', 'split0_test_score',...
'split2_test_score', ...
'std_fit_time', 'std_score_time', 'std_test_score']

Attributes

cv_results_ : dict of numpy (masked) ndarrays
A dict with keys as column headers and values as columns, that can be imported into a pandas DataFrame.

For instance the below given table

+------------+-----------+------------+-----------------+---+---------+ |param_kernel|param_gamma|param_degree|split0_test_score|...|rank_t...| +============+===========+============+=================+===+=========+ | 'poly' | -- | 2 | 0.80 |...| 2 | +------------+-----------+------------+-----------------+---+---------+ | 'poly' | -- | 3 | 0.70 |...| 4 | +------------+-----------+------------+-----------------+---+---------+ | 'rbf' | 0.1 | -- | 0.80 |...| 3 | +------------+-----------+------------+-----------------+---+---------+ | 'rbf' | 0.2 | -- | 0.93 |...| 1 | +------------+-----------+------------+-----------------+---+---------+

will be represented by a cv_results_ dict of:

{
'param_kernel': masked_array(data = ['poly', 'poly', 'rbf', 'rbf'],
mask = [False False False False]...)
'param_gamma': masked_array(data = [-- -- 0.1 0.2],
mask = [ True True False False]...),
'param_degree': masked_array(data = [2.0 3.0 -- --],
mask = [False False True True]...),
'split0_test_score' : [0.80, 0.70, 0.80, 0.93],
'split1_test_score' : [0.82, 0.50, 0.70, 0.78],
'mean_test_score' : [0.81, 0.60, 0.75, 0.85],
'std_test_score' : [0.01, 0.10, 0.05, 0.08],
'rank_test_score' : [2, 4, 3, 1],
'split0_train_score' : [0.80, 0.92, 0.70, 0.93],
'split1_train_score' : [0.82, 0.55, 0.70, 0.87],
'mean_train_score' : [0.81, 0.74, 0.70, 0.90],
'std_train_score' : [0.01, 0.19, 0.00, 0.03],
'mean_fit_time' : [0.73, 0.63, 0.43, 0.49],
'std_fit_time' : [0.01, 0.02, 0.01, 0.01],
'mean_score_time' : [0.01, 0.06, 0.04, 0.04],
'std_score_time' : [0.00, 0.00, 0.00, 0.01],
'params' : [{'kernel': 'poly', 'degree': 2}, ...],
}

NOTE

The key 'params' is used to store a list of parameter settings dicts for all the parameter candidates.

The mean_fit_time, std_fit_time, mean_score_time and std_score_time are all in seconds.

For multi-metric evaluation, the scores for all the scorers are available in the cv_results_ dict at the keys ending with that scorer's name ('_<scorer_name>') instead of '_score' shown above. ('split0_test_precision', 'mean_train_precision' etc.)

best_estimator_ : estimator
Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data. Not available if refit=False.

See refit parameter for more information on allowed values.

best_score_ : float
Mean cross-validated score of the best_estimator

For multi-metric evaluation, this is present only if refit is specified.

This attribute is not available if refit is a function.

best_params_ : dict
Parameter setting that gave the best results on the hold out data.

For multi-metric evaluation, this is present only if refit is specified.

best_index_ : int
The index (of the cv_results_ arrays) which corresponds to the best candidate parameter setting.

The dict at search.cv_results_['params'][search.best_index_] gives the parameter setting for the best model, that gives the highest mean score (search.best_score_).

For multi-metric evaluation, this is present only if refit is specified.

scorer_ : function or a dict
Scorer function used on the held out data to choose the best parameters for the model.

For multi-metric evaluation, this attribute holds the validated scoring dict which maps the scorer key to the scorer callable.

n_splits_ : int
The number of cross-validation splits (folds/iterations).

refit_time_ : float
Seconds used for refitting the best model on the whole dataset.

This is present only if refit is not False.

Notes

The parameters selected are those that maximize the score of the left out data, unless an explicit score is passed in which case it is used instead.

If n_jobs was set to a value higher than one, the data is copied for each point in the grid (and not n_jobs times). This is done for efficiency reasons if individual jobs take very little time, but may raise errors if the dataset is large and not enough memory is available. A workaround in this case is to set pre_dispatch. Then, the memory is copied only pre_dispatch many times. A reasonable value for pre_dispatch is 2 * n_jobs.

Parameters

penalty : {'l1', 'l2', 'elasticnet', 'none'}, default='l2'
Used to specify the norm used in the penalization. The 'newton-cg', 'sag' and 'lbfgs' solvers support only l2 penalties. 'elasticnet' is only supported by the 'saga' solver. If 'none' (not supported by the liblinear solver), no regularization is applied.

dual : bool, default=False
Dual or primal formulation. Dual formulation is only implemented for l2 penalty with liblinear solver. Prefer dual=False when n_samples > n_features.

tol : float, default=1e-4
Tolerance for stopping criteria.

C : float, default=1.0
Inverse of regularization strength; must be a positive float. Like in support vector machines, smaller values specify stronger regularization.

fit_intercept : bool, default=True
Specifies if a constant (a.k.a. bias or intercept) should be added to the decision function.

intercept_scaling : float, default=1
Useful only when the solver 'liblinear' is used and self.fit_intercept is set to True. In this case, x becomes [x, self.intercept_scaling], i.e. a "synthetic" feature with constant value equal to intercept_scaling is appended to the instance vector. The intercept becomes intercept_scaling * synthetic_feature_weight.

Note! the synthetic feature weight is subject to l1/l2 regularization as all other features. To lessen the effect of regularization on synthetic feature weight (and therefore on the intercept) intercept_scaling has to be increased.

class_weight : dict or 'balanced', default=None
Weights associated with classes in the form {class_label: weight}. If not given, all classes are supposed to have weight one.

The "balanced" mode uses the values of y to automatically adjust weights inversely proportional to class frequencies in the input data as n_samples / (n_classes * np.bincount(y)).

Note that these weights will be multiplied with sample_weight (passed through the fit method) if sample_weight is specified.

random_state : int, RandomState instance, default=None
The seed of the pseudo random number generator to use when shuffling the data. If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by np.random. Used when solver == 'sag' or 'liblinear'.

solver : {'newton-cg', 'lbfgs', 'liblinear', 'sag', 'saga'}, default='lbfgs'

Algorithm to use in the optimization problem.

For small datasets, 'liblinear' is a good choice, whereas 'sag' and
'saga' are faster for large ones.
For multiclass problems, only 'newton-cg', 'sag', 'saga' and 'lbfgs'
handle multinomial loss; 'liblinear' is limited to one-versus-rest schemes.
'newton-cg', 'lbfgs', 'sag' and 'saga' handle L2 or no penalty
'liblinear' and 'saga' also handle L1 penalty
'saga' also supports 'elasticnet' penalty
'liblinear' does not support setting penalty='none'

Note that 'sag' and 'saga' fast convergence is only guaranteed on features with approximately the same scale. You can preprocess the data with a scaler from sklearn.preprocessing.

max_iter : int, default=100
Maximum number of iterations taken for the solvers to converge.

multi_class : {'auto', 'ovr', 'multinomial'}, default='auto'
If the option chosen is 'ovr', then a binary problem is fit for each label. For 'multinomial' the loss minimised is the multinomial loss fit across the entire probability distribution, *even when the data is binary*. 'multinomial' is unavailable when solver='liblinear'. 'auto' selects 'ovr' if the data is binary, or if solver='liblinear', and otherwise selects 'multinomial'.

verbose : int, default=0
For the liblinear and lbfgs solvers set verbose to any positive number for verbosity.

warm_start : bool, default=False
When set to True, reuse the solution of the previous call to fit as initialization, otherwise, just erase the previous solution. Useless for liblinear solver. See the Glossary <warm_start>.

n_jobs : int, default=None
Number of CPU cores used when parallelizing over classes if multi_class='ovr'". This parameter is ignored when the solver is set to 'liblinear' regardless of whether 'multi_class' is specified or not. None means 1 unless in a joblib.parallel_backend context. -1 means using all processors. See Glossary <n_jobs> for more details.

l1_ratio : float, default=None
The Elastic-Net mixing parameter, with 0 <= l1_ratio <= 1. Only used if penalty='elasticnet'. Setting l1_ratio=0 is equivalent to using penalty='l2', while setting l1_ratio=1 is equivalent to using penalty='l1'. For 0 < l1_ratio <1, the penalty is a combination of L1 and L2.

Attributes

classes_ : ndarray of shape (n_classes, )
A list of class labels known to the classifier.

coef_ : ndarray of shape (1, n_features) or (n_classes, n_features)
Coefficient of the features in the decision function.

coef_ is of shape (1, n_features) when the given problem is binary. In particular, when multi_class='multinomial', coef_ corresponds to outcome 1 (True) and -coef_ corresponds to outcome 0 (False).

intercept_ : ndarray of shape (1,) or (n_classes,)
Intercept (a.k.a. bias) added to the decision function.

If fit_intercept is set to False, the intercept is set to zero. intercept_ is of shape (1,) when the given problem is binary. In particular, when multi_class='multinomial', intercept_ corresponds to outcome 1 (True) and -intercept_ corresponds to outcome 0 (False).

n_iter_ : ndarray of shape (n_classes,) or (1, )
Actual number of iterations for all classes. If binary or multinomial, it returns only 1 element. For liblinear solver, only the maximum number of iteration across all classes is given.

Notes

The underlying C implementation uses a random number generator to select features when fitting the model. It is thus not uncommon, to have slightly different results for the same input data. If that happens, try with a smaller tol parameter.

Predict output may not match that of standalone liblinear in certain cases. See differences from liblinear <liblinear_differences> in the narrative documentation.

References

L-BFGS-B -- Software for Large-scale Bound-constrained Optimization
Ciyou Zhu, Richard Byrd, Jorge Nocedal and Jose Luis Morales.
http://users.iems.northwestern.edu/\~nocedal/lbfgsb.html

LIBLINEAR -- A Library for Large Linear Classification
https://www.csie.ntu.edu.tw/\~cjlin/liblinear/

SAG -- Mark Schmidt, Nicolas Le Roux, and Francis Bach
Minimizing Finite Sums with the Stochastic Average Gradient
https://hal.inria.fr/hal-00860051/document

SAGA -- Defazio, A., Bach F. & Lacoste-Julien S. (2014).
SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives
https://arxiv.org/abs/1407.0202

Hsiang-Fu Yu, Fang-Lan Huang, Chih-Jen Lin (2011). Dual coordinate descent
    methods for logistic regression and maximum entropy models.
    Machine Learning 85(1-2):41-75.
    https://www.csie.ntu.edu.tw/\~cjlin/papers/maxent_dual.pdf

Examples

>>> from sklearn.datasets import load_iris
>>> from sklearn.linear_model import LogisticRegression
>>> X, y = load_iris(return_X_y=True)
>>> clf = LogisticRegression(random_state=0).fit(X, y)
>>> clf.predict(X[:2, :])
array([0, 0])
>>> clf.predict_proba(X[:2, :])
array([[9.8...e-01, 1.8...e-02, 1.4...e-08],
[9.7...e-01, 2.8...e-02, ...e-08]])
>>> clf.score(X, y)
0.97...

Код

Текст

Enter to Rename, ⇧Enter to Preview

1/1

def cross_val_score(estimator, X, y=None, groups=None, scoring=None, cv=None, n_jobs=None, verbose=0, fit_params=None, pre_dispatch='2*n_jobs', error_score=np.nan)